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[57] ABSTRACT 

A synchronotis voice/data messaging system allows parties 
to communicate both voice and graphic information or other 
data interactively or to retrieve such information synchro- 
nously in the exact sequence in which it was created. Voice 
information and data information are transmitted on a voice 
channel and a data channel. Time stamps are generated to 
synchronize the voice and data information. The time stamps 
can be generated either in a user device or in a multimedia 
messaging system. In either case, the voice and graphic or 
other data is played back to the called party in the exact 
sequence and at a rate corresponding to the rate at which it 
was received. Such a system can also be used to provide 
other services such as synchronous voice/data messaging 
while on hold or as part of a multimedia greeting. 

6 Claims, 3 Drawing Sheets 
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SYNCHRONOUS VOICEVDATA MESSAGE 
SYSTEM 

This application is a continuation of application Sen No. 
08/635,210, filed Apr. 17, 1996, now abandoned which is a 
continuation of application Ser. No. 08/255,770, filed Jun. 7, 
1994 DOW abandoned, 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The invention concerns telecommunications devices use- 
ful for synchronous voice data messaging. In particular, the 
invention concerns devices for transmitting and receiving 
both voice and display information and includes portable 
devices such as personal digital assistants. 

2. Related Art 

Usually, the simplest way for persons to convey informa- 
tion to each other is with the aid of a pencil and paper. 
Typically, a concept or idea is described verbally while 
elements being described are sketched, drawn or written on 
paper. Modem work stations, including pen-based work 
stations, allow similar interaction to take place between 
persons in remote locations. Parlies to a discussion can share 
a graphical/visual space where they can make and modify 
sketches and drawings at the same time. 

A difBculty is encountered in systems where such infor- 
mation is to be transferred in a non-real-time fashion. For 
example, in a system in which voice and data, such as 
graphic information, are stored for later playback, the 
graphic information is generally stored in its final form. 
When retrieved and reproduced, the viewer sees only the 
final graphic product and loses the sequence in which the 
graphic information, such as a drawing, was constructed. As 
a result, one who simultaneously listens to the voice mes- 
sage and views the graphic message at a later time loses the 
information inherent in viewing the sequence in which the 
graphic message was created. Thus, there is a need for a 
voice/data messaging system which preserves the synchro - 
nicity of voice and data information. To date, no such 
systems are available. 

SUMMARY AND OBJECTS OF THE 
INVENTION 

In view of the above limitations of the related art it is an 
object of the invention to provide an apparatus which allows 
parties to share voice and graphical space interactively. 

It is a further object of the invention to provide a party 
with graphical or other data information synchronously with 
voice information. 

It is still another object of the invention to allow such 
sharing of voice and graphical space interactively and to 
provide such graphical and voice information synchronously 
over a telecommunications link. 

It is still another object of the invention to provide these 
features with portable devices. 

It is still a further object of the invention to allow 
playback of voice and data messages in a synchronous 
manner at any time after the message is generated, for 
example messages including graphic, voice information or 
other drawing information. 

A system for synchronous data and voice storage and 
retrieval according to the invention includes a voice com- 
munication device to provide voice information from a 
transmitting user to a voice storage device, such as a voice 
recording device, under the control of a service controller 
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Such a system also includes a data communications device 
which provides data from the transmitting user to a data 
storage device, such as a data recording device, also under 
the control of a service controller. In a system according to 

5 the invention, the service controller includes a time stamp- 
ing means, such as a clock, counter or other time indicator, 
which synchronizes data information with the voice infor- 
mation as the voice information and the data information are 
generated and/or stored. This is accomplished by generating 

10 data and voice synchronizing time stamps. 

According to the invention, a playback device plays back 
the data information and the voice information in a sequence 
and at a rate which matches the sequence in which the data 
and voice information were generated and/or stored as 

15 determined from the time stamps. Thus, the system allows 
data information representing written information generated 
in real time synchronously with voice information to be 
reproduced in the same sequence and at the same rate in 
which it was generated. 

A system according to the invention also has controls to 
provide a pause in voice recording while data recording is 
continued. This featiu^e might be employed when a user is 
sketching a diagram to be transmitted as data, while not 
speaking for some time period. Such a pause could be 
implemented automatically when voice data is not detected 
for a predetermined time period or manually by activation of 
the feature by the transmitting user. 

A system according to the invention would also allow a 
user to perform certain functions. Control signals to imple- 
ment various commands and other instructions at appropri- 
ate times could be stored. For example, by activating a key 
or some other means to execute a predetermined stored 
command, a user could clear a screen display. 

35 The playback device may have auxiliary controls to 
customize playback functions including, for example, freeze 
frame, rewind, fast -forward, pause and other typically desir- 
able playback functions. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will be described in particularity with 
reference to the drawings in which: 

FIG. 1 illustrates a system in which the invention can be 
employed; 

FIG. 2 is an overall block diagram of a system according 
to the invention; 

FIG. 3 illustrates one embodiment according to the inven- 
tion; and 

50 FIG. 4 iQustrates an alternative embodiment of a system 
according to the invention. 

DETAILED DESCRIPTION OF THE 
INVENTION 

55 FIG. 1 illustrates a system 100 in which a system accord- 
ing to the invention can be implemented. First voice device 
102 communicates with network 120 over a voice channel 
104. Network 120 includes a controller 122 and switch 124 
to route the voice information on voice link 154 to voice 

60 device 152. The links 104, 108, 154, 158 in the system are 
shown to be bidirectional and the system is intended to 
encompass communications originating in any of devices 
102, 106, 152, and 156. In the case where the sender, for 
example the user of voice device 102, wishes to develop a 

65 sketch or drawing while discussing a matter with the party 
receiving the call, the sender may use a device 106, such as 
a personal digital assistant (PDA), computer, or other device 
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which provides data, such as graphic or text data. According 
to the invention, as the voice information is transmitted from 
voice device 102 on voice link 104 to the network 120, the 
simuhaneously generated graphical information is transmit- 
tcd from device 106 on data link 108 to network 120. 
Network 120 processes the voice and data information 
received and transmits the voice information to voice device 
152 over voice link 154 and the graphical or other data 
information to data device 156 over data link 158. This voice 
and data information is also transmitted simultaneously in a 
synchronous manner. As the sender provides voice informa- 
tion and simultaneously created graphical information, the 
voice and graphical information is received in the same 
sequence at the recipient's voice and data apparatus 152 and 
156, respectively. The recipient can also be using a PDA, 
computer or other device which utilizes data, such as graphic 
or text data. As previously noted, the system works in the 
same manner in the other direction for communications 
originating at voice and data devices 152, 156 and destined 
for voice and data devices 102, 106. It is also possible to 
integrate the functions of the voice and data devices into a 
single device. 

In another mode of operation according to the invention, 
in systems which provide a time delay between transmission 
and playback or in systems where playback is at a later time, 
it is not necessary for the voice and data to be transmitted 
synchronously by the user. It is only necessary that coding 
be provided, such that the correct relative timing and order 
sequence of the voice and data can be reconstructed to 
provide synchronous playback of voice and data informa- 
tion. Where the voice information is to be routed to voice 
mail and data routed to data storage, for example, where the 
originator receives no answer to a call and voice mail is 
activated, data link resources can be conserved by using the 
same link, e.g., 108, to transmit both the voice information 
to voice mail and the data information to data storage. This 
is possible since the coding provides the correct timing and 
order sequence of the information for playback. According 
to the invention, information can also be transmitted to a 
PDA for synchronous voice and data playback using a single 
transmission link. The coded information can be decoded in 
the PDA after both the voice and data have been received 
and synchronous playback can be performed. 

A processor is used to generate time stamps that synchro- 
nize the voice and data information transmitted on voice link 
104 and data link 108. Such time stamps can represent 
absolute time or time measured from a fixed event, such as 
a call connection, and can be generated in any known 
manner, for example by a clock, counter or other means for 
measuring absolute or elapsed time. These time stamps are 
available to be transmitted through voice link 154 and/or 
data link 158 to voice apparatus 152 and data/or apparatus 
156, which, for example, could be a display apparatus with 
a di^lay screen and/or memory. The processor for gener- 
ating such time stamps can reside in network 120, for 
example, in controller 122, or in the data generation device, 
e.g., device 106. In one embodiment voice data from device 
102 is generated, recorded and played back in a conventional 
manner, while data from device 106 is played back in the 
order and time dictated by the time stamps to achieve 
synchronicity between the data and voice information. 
Where voice information is sampled at a fixed rate, time 
information is inherent and time stamps for the voice 
information are not absolutely required. 

However, time stamps for voice information are required 
where voice compression techniques, such as deleting 
silence, are used. This is necessary to assure that when the 
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compressed voice information is played back it is played 
back in accordance with the time stamps to synchronize the 
voice and data information. Thus, in another embodiment, 
both the data and voice information are time stamped either 

5 upon receipt by network 120 or upon generation of the voice 
and data by time stamping circuitry in the originating data 
device, e.g., 106 and transmitting voice device, e.g., 102. 
Another application for time stamping voice information 
was previously discussed where one form of information, 
such as voice information is sent first and the other form of 
information, such as data, is sent at a later time, for example, 
using the same link. In this case, coding both the voice and 
data information with time stamps conserves system link 
resources, since the time stamps can be used to synchronize 
the voice and data information on playback. In addition, 

15 where link resources are scarce, as when the system in which 
the communication takes place is very busy, time stamping 
the data allows the data to be transmitted on a link that 
becomes available during a later period of reduced load, 
since the voice and data information can be synchronized for 

20 playback using the time stamps. 

Synchronous playback would then be achieved by play- 
back of both voice and data information at the corresponding 
time stamps. Synchronous playback can be achieved by 
synchronously transmitting the voice and data information 

25 from network 120 to data device 156 and voice device 152 
in accordance with the time stamps. Devices 152 and 156 
can then store or playback the voice and data information in 
the time ordered sequence and at the same rate in which it 
was received. In this case, it is not absolutely necessary for 

30 devices 152 and 156 to receive the time stamps. 
Alternatively, the raw voice and data information can be 
transmitted from network 120 with the time stamp informa- 
tion in any order and devices 152 and 156 can reconstruct 
the time ordered sequence for playback according to the 

35 time stamps received. These alternatives are by way of 
illustration and not limitation, as a system according to the 
invention can employ any arrangement wherein the voice 
and data information can be played back synchronously in 
the time ordered sequence and at the rate in which such data 

40 was generated. A system according to the invention can also 
implement a fast or slow playback mode in which the voice 
and data are played back synchronously in the same time 
ordered sequence at which the information was received, but 
at a rate faster or slower than the rate at which the infor- 

45 mation was received. 

When the designated recipient of the caller's voice and 
data information is unavailable, either because the recipient 
does not answer or the recipient's voice and data device are 
otherwise busy, network 120 can implement the recipient's 

50 busy/don't answer response. For example, network 120 
could send the time stamped voice and data information to 
a voice mail or other recording apparatus. By accessing the 
stored voice and data information with the time stamps, the 
intended recipient can play back the exact sequence in which 

55 the voice and data information were transmitted. Following 
the exact sequence and rate using the time stamps, the 
display apparatus illustrates the generation of the graphic 
information synchronously with the corresponding voice 
information. Thus, the recipient receives the benefit of the 

60 inherent information conveyed by seeing the sequence of the 
creation of the graphic image with the corresponding voice 
data. This is in contrast to a typical system in which the 
graphical information will initially be displayed in its com- 
plete form and the voice information will be played back 

65 independently as an associated voice annotation. 

Even in cases where the recipient receives the voice and 
data information instantly, the time stamps can still be used 
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by the recipient to record the voice and data iofonnation for information, respectively, to communication device B. This 

later playback in a synchronous manner, as discussed above. would allow the communication device B to receive and 

FIG. 2 further illustrates an embodiment of a system record the time stamps themselves. With the time stamped 

acxx)rding to the invention, which can be implemented in a information, communication device B could synchronously 
conventional wire line telephone system, a cellular tele- 5 reproduce the voice and data information in the sequence 

phone system, or any other kind of telecommunication and timing in which it was generated. Unless voice com- 

system. Telecommunications device A 202 has a record and pression is employed, it is not necessary to lime stamp voice 

playback feature 204 and communicates with a computer or information, since time stamps are inherent in voice infor- 

databasc 208. The computer/database 208 and record/ nation sampled at a fixed rate. 

playback features can be directly integrated with communi- Another feature according to the invention is the ability to 

cation device A 202 or may be embodied in a separate provide a template or other graphic information on which 

apparatus communicating with communications device A new graphic information is superimposed. For example, the 

202. user of communication device A 202 could access data base 

Communications device A 202 provides voice and data 208 for a template and superimpose additional graphic 

information over links, 210, 212 to a switching office, for information on the template. The data base could be inte- 

example, central ofBce (co) 214. Switching ofEce 214 grated into or separate from the communication device A. 

includes routing switch 216 responsive to service controller Th^ template would be transmitted as data. The graphic 

218 for routing voice information. Data information is information superimposed thereon would also be transferred 

routed to the service conUt^Uer 218. As previously noted, as data synchronously with the voice information generated 

data information may be received with time stamps gener- at the same time the new graphic information is superim- 

atcd at the originating device or data may be time stamped posed or created. 

upon receipt at the service controller 218. Service controller FIG. 3 illustrates a wireless application of a system 

218 provides switch commands on signal lines 220 to switch according to the invention. In a typical scenario, a wireless^^-— 

216. In addition, service controller 218 also routes voice subscriber originates a call under the control of personal \ 

mail control information to voice mail unit 224 on lines 222, digital assistant (PDA) 301 through base station 303 which 

which receives voice information from the switch via line is controlled by base station controller 305. The base station 

221. Optionally the voice information or the voice mail controller 305 provides information to a mobile switching 

control information may have time stamps, as discussed center 307 which is connected to the telephone network and 

above. to a central office switch 309. At the central office (CO) 311, 

Tune stamps may be generated in response to voice and ^° the central office switch 309 is also connected to a service 

data information or may be generated continuously, for conU-ol point 313, which can provide short message service 

example, as indications of a real time clock. As previously 315 (SMS) and connect to a data packet network along 

noted, it may not be necessary to time stamp both voice and signal lines 317. An intelligent peripheral 318 can also be 

data information, as long as at least one of these types of data employed in the central office receiving and transmitting 

is coded sufficiently to provide synchronous playback control information on signal line 316. 
according to the time ordered sequence of voice and data In such an application, the PDA informs the service 

generation. control point of a number to be called. The service control 

The time stamps can be handled solely in the service point originates the call on behalf of the user as soon as the 

controller, if the voice and data information is routed within user goes off hook on the associated phone, such as portable 

the central office, for example to a voice mail system 224. In 319. When the called party is busy or does not answer, the 

this instance, only circuitry in the switching office 214 need user or personal digital assistant 301 and portable 319 can 

keep track of the time stamps and the communication leave a voice message or elect to leave a synchronous 

devices 202 and 252 need not be aware of their existence. In voice/data message as previously described. Data from the 
this case, when communication device B 252 later accesses 45 PDA could be sent using the second B channel associated 

voice and data information from communication device A with the voice B channel in an integrated services digital 

202 through recording subsystem 224, such as voice mail, network (ISDN) environment. 

outgoing voice information is provided from switch 216 on A synchronous voice/data messaging application pro- 
signal lines 232 to communication device B. Synchronously, vided on the personal digital assistant 301 collects data 
service controller 218 provides data infonnation on signal samples from the pen input and transmits this information 
line 230 to communication device B. The user at commu- along the data channel. The service control point, for 
nication device B will then receive synchronized voice and example, a multimedia mail system, time stamps these 
data information to produce a graphic image in the order and samples and stores them synchronously with the voice 
timing in which it was created synchronously with the voice information. When retrieved, the service control point sends 
description. ^5 raw samples to the synchronous voice/data messaging appli- 
Alternatively, communication device B could be equipped cation in the PDA of the called party following the order and 
with record and playback feamres 254 and a database 256, time of the time stamps, together with the voice information 
as previously discussed with respect to communication to the associated phone. In this way, desired synchronicity is 
device A. By receiving the information in the time sequence achieved without the need for sending time stamps to the 
in which it was generated, record/playback device 254 can 50 called party. Because the service control point 313 is the 
be used to record the information in the same sequence. only entity dealing with time stamps, the PDA messaging 
Thus, user of communication device B 252 has a complete application can also be tised for interactive sharing of the 
record of the sequence and corresponding voice information visual space between the PDA of the caller and the called 
which the user of communication device A employed to party in real time. 

provide the information. 65 FIG. 4 illustrates a further embodiment of a system 

Another alternative is for the switch 216 and service according to the invention. In this case, personal digital 

controller 218 to transmit voice and time stamped data assistant 401 provides information to base station 403 and, 
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from base station controller 405, to mobile switching center 
407. In this case, the mobile switching center 407 incorpo- 
rates an advanced intelhgent network to perform the switch- 
ing functions. The advanced intelUgeot network communi- 
cates with a service control point/adjunct 408 which 
provides information to the data packet network on signal 
lines 409 and to short message service 411 and to voice 
service 410, which receives control information from mobile 
switch center 407 on signal line 412. Intelligent peripheral 
414 also communicates with mobile switch center 407. 
Portable 413 corresponds to portable 319 in FIG. 3. Opera- 
tion of a system configured in this way is essentially the 
same as that previously described herein. 

A synchronous voice/data messaging system according to 
the invention also provides the opportunity to enhance 15 
services. For example, voice messaging is a commonly used 
mechanism to leave messages with another party without the 
intervention of a third party, such as a secretary. For sub- 
scribers that have a PDA associated with their phone, a 
synchronous voice data message can be left, as described 20 
herein. 

In addition, when a calHng party is routed to another 
party's multimedia mail system, the receiving party can 
provide a synchronous/data message greeting. The called 
party's synchronous/data message greeting can be person- 
alized or otherwise customized. For example, the called 
party can transmit a synchronous voice/data message with 
soft keys for the caller to indicate the urgency of the call or 
whether the caller is to be reached on a particular device 
such as a wireless system, or other information, 

A synchronous voice/data greeting can also be used to 
provide advertising. For example, when a call to a business 
is routed to a multimedia mail system or alternative call 
distribution (ACD) queue, a brief description of the 
business, its services and other relevant information can be 
provided as a synchronous voice/data message before 
requesting the caller to respond or leave a message. 

A further service which can be offered includes advertis- 
ing while a caller is on hold. During these short periods of 
time, a multimedia system can be used to provide an 
effective synchronous voice/data advertising message to the 
caller on hold. 

A system according to the invention also has application 
in education and business where real time presentations with 45 
voice and graphic information are transmitted to other 
locations and played either contemporaneously or at a later 
time. For example, synchronous playback of voice informa- 
tion and graphical information in the sequence in which the 
voice and data information was generated using a system 50 
according to the invention described herein reduces the need 
for sophisticated video equipment and wide bandwidth 
transmission channels. 

While several embodiments of the invention have been 
described, it will be understood that it is capable of fiJrther 55 
modifications, and this application is intended to cover any 
variations, uses, or adaptations of the invention, following in 
general the principles of the invention and including such 
departures from the present disclosure as to come within 
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first, second, third and fourth teleconununications links, 
a first portable communications device comprising a voice 
input unit, a data input unit, and a transceiver unit, 
wherein the data input unit accepts data as input by a 
user command from a data base, the voice input unit 
accepts voice signals of a user, and the transceiver unit 
transmits said voice signals over the first telecommu- 
nications link and the data over said the second tele- 
communications link, 
a central switching oflSce for receiving and storing said 
voice signals transmitted over the first telecommunica- 
tions link from said first portable communications 
device and the data U-ansmitted over the second tele- 
communications link from said first portable commu- 
nications device, and retransmitting said data over said 
third telecommunications link and retransmitting said 
voice signals over said fourth telecommunications link, 
said central switching office including: 
a service controller for receiving the data directly from 

the first portable communications device over the 

second telecommunications link, 
a switch for receiving the voice signals directly from 

the first portable communications device over the 

first telecommunications link, and 
a voice mail unit connected to the service controller and 

the switch and configured to temporarily store the 

voice signals in a case where a destination device is 

not currently available to receive the data and the 

voice signals, 

a second portable communication device, which corre- 
sponds to the destination device, said second portable 
communication device comprising a voice output unit, 
a data display unit, and a transceiver unit, wherein said 
second portable communication device receives said 
voice signals over the fourth communications link and 
said data over third telecommunications link, the data 
display unit outputs the data received over said third 
telecommunications link as a dynamic visible display 
and the voice output unit outputs the voice received 
over said fourth telecommunications link as audible 
voice instructions corresponding to the dynamic visible 
display, and 

a time stamp means for placing time stamps on the voice 
signals and the data, 

wherein the switch outputs the voice signals received over 
the first telecommunications link to the fourth telecom- 
munications link based on control signals received 
from the service controller, the service controller pro- 
viding the control signals to the switch based on 
decoding of the data received over the second telecom- 
munications link. 

2. A personal communications and multimedia mail sys- 
tem according to claim 1, wherein said time stamp means is 
disposed in one of the first portable telecommunications 
device and the service controller. 

3. A personal communications and multimedia mail sys- 
tem according to claim 2, wherein, when said time stamp 
means is disposed in said service controller, said time stamp 
means places said time stamps on said data received over 



knowledge or customary practice in the art to which the 60 said second telecommunications lirik based on a time when 



invention pertains, and as may be applied to the essential 
features hereinbefore set forth and falling within the scope 
of the invention or the limits of the appended claims. 
What is claimed is: 

1. A personal communications and multimedia mail sys- 
tem for providing synchronous commxmication of data and 
voice, comprising: 



said data are received by said service controller over said 
second telecommunications link, and said time stamps are 
placed on said voice signals received over said first tele- 
communications link by said switch based on said control 
55 signals sent from said service controller to said switch. 

4. A personal communication and multimedia mail system 
according to claim 1, wherein said second portable telecom- 
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miinications device records the time stamps based' on the 
data received over the third telecommunications hnk and the 
voice signals received simultaneously over the fourth tele- 
communications link, and 

wherein the second portable telecommunications device 
is capable of synchronously reproducing the voice 
signals and the data at a later time based on the time 
stamps. 

5. A personal communication and multimedia mail system 
according to claim 1, wherein in a case where said voice 
mail unit has temporarily stored the voice signals due to said 
destination device not being available to receive the data and 
the voice signals at a lime when the data and the voice 
signals were initially received by the central switching 
office J at a later time when the destination device is able to 
receive the data and the voice signals based on a signal sent 
from the destination device to the central switching office, 
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the temporarily stored voice signals are output to said switch 
and routed to said destination device over the fourth tele- 
communications link, and at a same time the data are routed 
to said destination device directly from said service control- 
ler over the third telecommunications link. 

6. A personal communication and multimedia mail system 
according to claim 1, wherein the data that is received by the 
service controller over the second telecommunications link 
are capable of being received at a different time with respect 
to when corresponding voice signals are received by the 
switch over the first telecommunications hnk, and 
wherein the time stamp means provides mformation so as 
to equate portions of the data to be output with portions 
of the voice signals at a synchronized timing by the 
second portable communications device. 
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